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MANAGEMENT AND OVERALL APPROACH 


Be oat. : . . are a TI ys retreat tal got . Jets ’ : ‘ ' . Pe ay ek ok gy et a ooo : , 7 eee ey 


‘ . 
-toe 1 a . 1 


wat : 5G . ‘| 
: ty ; i? Om gn a ea Wee et ‘ ‘ : ' ; . : 2 wm sey tere She ans : . : cee, ' 4 
tao iin pie ee ie ge “dewctonsat aa nies ‘4 — whe Ve: exntosd ety rats Bane iil lta fed tore otal tat eet tad thes ceed haere od vin nied wesc sed eed deel tes es » Age ert 











wae 
. 

any 
- 4 


eras, 
Wey ak 


FER PTT 
a 4 


Benes 
iu atte 
at be 


Part lil. 


MANAGEMENT AND OVERALL APPROACH 


To summarize the lessons learned from ABM 
R&D is challenging, because the period of time 
was long, the breadth of research wide, and the 
follow-on development program immense. This 
part of the report discusses the most impor- 
tant lessons learned about the management and 
overall approaches used in a project such as 
this. The discussion is discursive and makes 


no attempt to present detailed supporting data 


or arguments. Every member of Bell Labora- 
tories Divisions 65 and 66 management contri- 
buted to this report by suggesting items, writing 
portions, and/or criticizing early drafts. There 
is a reasonable consensus on the views ex- 
pressed here, but probably not on their relative 
importance. 


Because this was a very big project, parti- 
cularly the SAFEGUARD portion, the lessons 
are most directly applicable to other very large 
projects and many will probably also apply to 
smaller programs. There has been no effort to 
analyze the effect of project size or to modify 
the conclusions for different conditions. 


Under the prime contract with Western 
Electric, Bell Laboratories had overall re- 
sponsibility for essentially all R&D work. Since 
Bell Laboratories has its own special organiza- 
tional characteristics, many of the statements 
made here have probably been affected by those 
characteristics. There has been no attempt to 


determine just how these characteristics have 
affected the conclusions or how the conclusions 
might be modified by different organizations. 
However, the close relationship between Bell 
Laboratories and Western Electric was exploited 
continuously during development and production. 


| OVERVIEW 


Feasibility 


Perhaps the most important lesson of this 
experience, and particularly of the SAFEGUARD 
part, is that development of a large weapons 
system can be completed on schedule to pre- 
scribed performance specifications, with effec- 
tively controlled costs. It is true that changes 
during development drastically cut down the 
overall deployment. Also, the overall system 
design was modified because of changes in ob- 
jectives and because of test results obtained 
during development. However, the development, 
including integration of the first installed site, 
proceeded on schedule and the system met the 
prescribed performance specifications. Cost 
performance is harder to determine because of 
inflation during the period, deployment changes, 
and lack of a design-to-cost objective. But it 
seems clear that costs were controlled effec- 
tively. 


Given the experience with some other im- 
portant weapons system developments (most of 
them smaller or less complex than this one) in 
failing to meet schedules, performance spe- 
cifications, or cost objectives, the ABM R&D 
effort is valuable as an example that such de- 
velopments can be successful. This is particu- 
larly important because one major argument 
against developing SAFEGUARD was that it 
could not be done successfully. 


Overall Approach 


One principle followed during the entire re- 
search and development period was to continu- 
ously evolve the system concept; that is, to re- 


peatedly synthesize and analyze systems that met — 


prescribed objectives against prescribed threats. 
Although most of the effort was spent on re- 
search and development of specific subsystems 
(radars, missiles, data processors, etc.), 
there were continuous studies of how subsys- 
tems could be used within a System. The sen- 
sitivity of system performance to subsystem 
specifications and to changes in objectives and 
threat was repeatedly studied. Designers con- 
centrated on aspects of subsystem performance 
most important to overall system performance. 


The overall approach might be summarized 
as plan, simulate, test — plan, simulate, 
test — plan, simulate, test. Every part of the 
development benefited from detailed planning. 
While there was little or no attempt to specify 
a planning format or technique, every subpro- 
ject manager was strongly encouraged to plan 
in detail the design, implementation, and test- 
ing of his piece of the system. Although these 
plans had to be revised many times, it was very 
beneficial to have them. To keep track of prog- 
ress, several methods of reporting were tried, 
as discussed later. 


Simulations were used on all levels of detail 
and for all parts of the design. In every case, 
simulations more than paid for themselves in 
either improved performance or reduced cost. 


When trouble was found, it was also discovered 
that more or better simulation would have pre- 
vented, or at least reduced, the difficulties. Be- 
cause the simulation results were important, 
there were extensive efforts to validate them with 
data from real tests. 


Perhaps the most important part of the over- 
all approach was testing. Effective testing de- 
pends strongly on detailed test specifications. 
Most of the development time and cost went into 
testing, and it always happened that even more 
tests were possible and, in fact, desirable. Con- 
sequently, test planning almost always consisted 
of selecting only the most essential tests from 
among those desirable. After each test, data 
analysis continued until any failures or anoma~ 
lies were understood. These comprehensive 
analyses, particularly of the Meck Island tests 
at Kwajalein Atoll, frequently resulted in signi- 
ficant system improvements. 


Constructing the prototype Missile Site Radar 
(MSR) at the Kwajalein test site made way for 
the Kwajalein field experiments, which were un- 
doubtedly the single most important step in 
achieving success. The project would have been 
even more successful if a more complete system 
prototype had been built. Building a prototype 
Perimeter Acquisition Radar (PAR) at a northern 
site early in the R&D program would have def- 
initely produced a better system design. There 
were major problems with developing the PAR 
software, mostly because no complete, con- 
sistent set of requirements could be checked out 
on a prototype PAR system. Without a prototype, 


_ the problems with requirements were not identi- 


fied clearly by the developers until the final de- 
velopment phase. No large project should be 
undertaken without allowing time and resources 
for constructing and extensively testing a com- 
plete prototype. 


The Kwajalein tests are perhaps better termed 
"field experiments," and they were almost un- 
believably successful. Successful tests meant 
not just meeting normal test objectives; they 
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produced data invaluable for completing system 
design. For instance, the data obtained on tank 
breakup were vital to the success of the project. 
Experiments were carried out in a carefully 
controlled environment (which helped both the 


. conduct and analyses of tests) rather than in the 


design threat environment only. It would have 
been impossible to obtain data for the final de- 
sign if the tests had only been trials against 
the design threat. 


In many other projects, separate organiza-~ 
tions within a company carry out their own field 
operations. A principal reason for the success- 
ful ABM field test program was the rotational 
assignment of systems, hardware, and software 
designers to the field sites. The reward was 
not only the transfer of design intent to the 
field tests but also the experience fed back to 
the design work when the designers returned to 
their laboratories, 


Probably second in importance only to the 
Kwajalein tests were the tests at the Tactical 
Software Control Site (TSCS) in Madison, N. J. 


Not only were these tests vital to software de- 


velopment, they were also an important source 
of data on the hardware and helped validate the 
system simulations. Later sections of this 
discussion expand on the importance of testing 
and the TSCS. 


The one constant factor throughout the pro- 
ject was the inevitability of change. At no time 
were the objectives, threat, design, or deploy- 
ment completely stable, and it was unrealistic 
to assume that any of them would remain fixed. 
Therefore, it was vital to be prepared for 
changes and to have effective procedures for 
controlling them. Making changes in hardware 
was a familiar problem from Bell Laboratories’ 
experience on Bell System and other military 
projects, and procedures for handling them 
were reasonably well established. But the 
complexity and high performance of the 
PAR and MSR systems necessarily pro- 
duced a greater number of changes during 
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testing and checkout than ever experienced be- 
fore with smaller systems of lesser performance. 


By careful planning, many changes were intro- 
duced into major subsystems — the MSR at 
Meck, the TSCS at Madison, the tactical 
MSR in North Dakota, and the PAR in North 
Dakota — during tests without materially af- 
fecting schedules. By scheduling test time among 
the many equipment users, changes could be 
introduced if the necessary drafting, shop, and 
quality assurance teams were available when re- 
quired. A mutual understanding between design- 
ers of hardware and designers of software test- 
ing routines, plus careful scheduling, normally | 
allowed enough time to physically disrupt hard- 
ware to introduce a change. Changes necessarily 
must be designed in small packages, but large 
packages of changes could be introduced because 
they were scheduled for periods when the system 
was down for a day or more and because work- 
crew schedules were kept flexible. After the 
changes, enough time was allowed for regres- 
sion testing of areas affected by the changes. 


By keeping the main technology fixed during 
the program, it was possible to produce, on 
schedule, equipment with proven reliable de- 
signs. On many other projects, hardware de- 
signs are continually changed to keep abreast of 
changing technology, schedules continually slip, 
and hardware is unreliable, 


While controlling hardware design changes 
was a big problem, controlling system and soft- 
ware changes was even more difficult because 
procedures were not as well established. It 
took a great deal of effort to develop techniques 
for controlling such changes. These techniques, 
plus strict discipline, resulted in a control pro- 
cedure without which the software development 
would have failed. (See Chapter 4 of Part II 
for a discussion of lessons learned in control- 
ling software changes. ) 


Overall system requirements were specified 
early in the development, and formal procedures 
for controlling changes in these requirements 


were immediately instituted. Software design 
was tied closely to these controlled system re- 
quirements. Even though there were many, 
many changes, this procedure was crucial to 
successful software development. 


Organization 


~~ 


One of the most important characteristics of 
this project was the interface between Bell 
Laboratories and the Army agency responsible 
for ABM development. Relationships were 
open, frank, and informal. Although formal 
contractual requirements were carefully nego- 
tiated and carried out, most of the information 
exchanges between the Army and the contractor 
were informal. Consequently, problems could 
be discussed as they came up and proper action 
taken immediately. Also, the significance of 
problems could be determined and the right 
priorities established. Little effort was wasted 
on insignificant problems. 


It was very important that the Army side of 
the interface be a single, responsible Program 
Manager. As with any very large project, there 
were many interfaces at various levels with the 
Army's SAFEGUARD organization both in Wash- 
ington and in Huntsville, Alabama. In addition, 
liaison was necessary with many Army organi- 
zations; e.g., nuclear effects, facilities re- 
quirements, communications. Despite the 
many interfaces, the SAFEGUARD System Of- 
fice and the SAFEGUARD System Command be- 
came an effective single agency responsible 
for resolving differences and adjudicating prob- 
lems whenever they occurred. 


Overall system changes were negotiated be- 
tween the System Requirements Department at 
Bell Laboratories and the corresponding or- 
ganization in the SAFEGUARD Project Office at 
Huntsville. Questions about system performance 
were discussed in proper context, and appro- 
priate formal action taken to change the system 
requirements. Subsystem development was 
thereby protected from many "what if" studies 
which could have delayed progress. 


This project, like others its size, took 
several companies to carry out, However, it 
was made clear that Bell Laboratories had over- 
all responsibility, and top managers in Bell 
Laboratories used their authority to make major 
decisions quickly. If responsibility had been 
split among several companies, or if other red 
tape had impeded Bell Laboratories managers, 
it is improbable that schedules could have been 
met and very probable that costs would have in- 
creased. Because Bell Laboratories could make 
decisions quickly, technical authority over sub- 
contractors could be delegated to lower levels 
when cost was not involved. This speeded up 
decision-making; it also increased motivation 
and inbred a sense of responsibility in the 
people working on the project. 


Interfaces between Bell Laboratories and the 
subcontractors received special attention. In 
very large and technically complex development 
projects, the importance of this cannot be 
overemphasized. Technical developments and 
problems in each area affect other areas. 
Whether the related developmental areas are 
within a single company or split among contrac- 
tor and subcontractor, almost continuous inter- 
change of information is essential. Monthly or 
quarterly technical reviews are inadequate un- 
less informal exchanges on the engineering level 
go on, by telephone and personal visits, at least 
once every week or two. With major subcon- 
tractors, it is often necessary to keep an en- 
gineering group at the subcontractor's plant. 
Without such frequent and informal engineer- 
ing level exchanges, the technical development 
would not have been as quick or successful as 
it was. 


In hardware subcontracting, enough time 
should be spent, before development starts, to 
work out a detailed design specification that both 
contractor and subcontractor fully understand. 
Requirements will change, so an early base for 
defining them is very important. This is par- 
ticularly true for a cost/schedule/performance 
incentive contract. 
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Attention to contractor/subcontractor re- 
lationships was particularly important with soft- 
ware development. Because it is impossible to 
state software requirements precisely until 
after system design is completed, it is very dif- 
ficult to employ subcontractors to develop com- 
plex software. If subcontractors are used, the 
interface must-be carefully specified and moni- 
tored. The large subcontractor effort used to 
develop SAFEGUARD software was divided into 
smaller, well-defined tasks that interfaced di- 
rectly with the Bell Laboratories first-line 
supervision. Tasks were rated monthly by the 
technical supervisor, and the rating results 
determined the final fee on the Cost-Plus-Award- 
Fee (CPAF) contract. This interface ensured 
responsive subcontractor performance at a man- 
ageable level on a very large project. These 
methods worked well, and when problems de- 
veloped it was usually where the customary in- 
terface procedures had not been used. 


Perhaps the most important principle fol- 
lowed in organizing the project was to install a 
top-quality manager over each subproject and 
give him full responsiblity for schedule, cost, 
and performance, The managers had broad and 
deep technical competence as well as managerial 
competence. Furthermore, they were expected 
to be aggressive in not just passively accepting 
a job definition and carrying it out. Instead, 
they actively explored interfaces between their 
subprojects and all others to anticipate problems, 
Although they used a variety of techniques to 
carry out their responsibilities, time after time 
their ability to recognize technical problems was 
fundamental in finding solutions to them. While 
it was certainly important to appoint the best 
general manager, it was absolutely vital that 
he have a high technical competence as well. 


SPECIFIC SUGGESTIONS 


This section contains comments less basic, 
but more specific, than those in the preceding 
sections. While the suggestions are grouped 


into general areas, each is largely independent 
of the others. 


Requirements 
System 


To design and develop a large, complex sys- 
tem in a few years, a large number of people is 
obviously required (several thousand at the peak 
for SAFEGUARD). To properly coordinate their 
efforts, a good set of requirements and/or spe- 
cifications is essential. For SAFEGUARD, the 
major specifications were: 

e Brief overall SAFEGUARD System speci- 

fications 


e Missile Site Equipment Subsystem speci- 
fication 


e Perimeter Acquisition Site Equipment Sub- 
system specification 


e SPARTAN Subsystem specification 
SPRINT Subsystem specification 

e Communication Equipment Subsystem spe- 

cification 

e Ballistic Missile Defense Center specifi- 

cation 

e Data Processing System Performance Re- 

quirements (DPSPR). 

The DPSPRs controlled software implementa- 
tion and actually specified the detailed system 
requirements. The following comments are 
based on experience with all of the specifica- 
tions, but mostly with the DPSPRs. 


A decision about the level of detail and the 
completeness of requirements must be carefully 
weighed. The decision depends on the relation- 
ship between the organization preparing the re- 
quirements and the organization that implements 
them, the resources available, the time avail- 
able, and various psychological factors. Obvi- 
ously, the closer the organizational relationship, 
the less need for completeness. On the other 
hand, if the organizations belong to different 
companies, requirements must be complete and 
detailed, It is probably better to err on the side 
of too much detail rather than too little. On 
SAFEGUARD, too few resources (about 7 percent) 
were allocated to preparing the requirements. 


Designers naturally object to overly detailed re- 
quirements because it limits them, and this im- 
portant, legitimate criticism should be care- 
fully guarded against. However, problems re- 
sulting from lack of detail can be even more 
serious, and therefore must be guarded against 
even more carefully. | 


It is vital that requirements be available as 
early in the program as possible. Those pre- © 
paring requirements must make decisions quick- 
ly, on the basis of incomplete analyses, and 
publish them in the shortest possible time. 

- Management should soften criticism of defects 
in the early issues of the requirements, because 
the defects inevitably result from pressure for 
early availability. 


Because objectives change, because problems 
become better understood, and because require- 
ments must be written quickly, changes are in- 
evitable. It is absolutely necessary to institute 
a way to control changes relatively early in the 
process. This change control must keep pro- 
posed changes from getting lost and ensure that 
every designer is working with a consistent and 
updated set of requirements. 


Requirements serve two purposes. They tell 
the customer what he is buying and the designer 
what he is designing. If these purposes conflict, 
the priority is to give the designer clear and 
unambiguous requirements. If necessary, give 
the customer explanatory material. 


To summarize: 


e Publish the first issue of requirements 
as early and as completely as possible. 
The first issue then forms a baseline 
for changes. 


Controlling changes in requirements is 
essential. The control system should 
become formal before the project is half 
completed. 


Carefully plan the degree of detail and 

completeness in the requirements. Do 

not allow unanalyzed resource restrictions 
-to determine these answers. 


Availability/ Reliability 


In a system like SAFEGUARD, a complete 
and easily understood availability and reliability 
budget is essential for each subsystem. This 
budget reflects the system effectiveness require- 
ments. In the real world of fixed resources and 
flexible system objectives, these requirements 
should be considered as objectives. As such, 
they should be reviewed carefully at intervals to 
ensure that the system stays in balance and that 
the cost of achieving the original objectives is 
clearly understood. In SAFEGUARD, for ex- 
ample, if system effectiveness is measured by 
the number of Minuteman ICBMs that survive an 
attack, a wide range in availability and reliability Po 
will produce acceptable results. The original eS 
objectives were set for the SENTINEL Area 
Defense System, which required more stringent 
requirements than the Minuteman defense. More He 
thorough consideration of possible reductions in | 
objectives might have lowered some requirements 
for components and subsystems. 


Man-Machine Subsystem 


The requirements originally specified for the 
CRT console, the lightpen, the teletypewriter, 
and other units in the Command and Control Dis- 
play System (CCDS) were far more extensive 
than necessary to support the man-machine re- 
lationship. Furthermore, it was found later that 
to supply many of these special capabilities 
required a software effort greater than the 
project could support. The man-machine inter- 
face was therefore greatly modified, resulting 
in two out of three available lightpen modes not 
being used. In addition, many CRT console ca- 
pabilities, such as the A-scope mode and lower 
case letters, are unused throughout the CCDS. At 
the beginning, a hard-nosed and realistic examina- 
tion of what man really had to do to control an 
ABM system, plus the assurance that require- 
ments were compatible with the capability of 
supporting software, would have drastically re- 
duced requirements on the Command and Control > 
System. The equipment would have been more 
economical to design, produce, and maintain. 
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Similarly, the user directed that many en- 
gagement control actions be included in the re- 
quirements because they were useful in anti- 
aircraft systems, even though they were in- 
effective for ABM systems. These actions in- 
cluded Hold Fire, Force Intercept, and Com- 
mand Destruct. If the requirements for ABM 
control actions had been agreed on sooner, the 
costs of designing, developing, and integrating 
display and control software would have been 
lower. While these problems did not signifi- 
cantly reduce capability of the man-machine 
subsystem, they definitely reduced the quality 
and "elegance" of the design. 


‘Nuclear Surety and Safety 


There should have been earlier, more effec- 
tive communication among design organizations 
and the government agencies responsible for 
nuclear surety and safety: the National Security 
Agency (NSA) and the Nuclear Weapons System 
Safety Committee (NWSSC). Prompt liaison 
could have avoided misunderstandings, mis- 
takes, and costly last-minute changes to the 
Nuclear Enable Authority and Launch Enable 


systems. 


The agencies began intensive final reviews 
after designs, which incorporated changes 
suggested during their preliminary reviews, 
had been released for manufacture. Under these 
circumstances, designers were understandably 
reluctant to accept the agencies’ comments and 
suggestions. After an additional year of review, 
NWSSC suggestions became a mandatory re- 
quirement for a new Launch Enable design con- 
cept. The designers regarded the new design 
and other practices imposed by the agencies as 
"ex post facto" requirements. Much cost, delay, 
and contention could have been avoided if: 

e All the organizations involved had better 

understood the roles of NSA and NWSSC 


and the strength of their requirements and 
recommendafions 


e Before design work started, the agencies 
had documented their requirements, pre- 
ferred concepts, and standards developed 
and imposed on other projects (e.g., use 


of nonsymmetrical connectors, omission 
of continuity loops, etc.) 


@ The above had been used as the basis for 
agency review with minimum application 
of criteria developed during review from 
"state of mind" guidelines 


@ The reviews had been more in parallel with 
design, to supply incremental concurrences 
from design through field test. 


Intelligence Data 


Gathering intelligence through U. S. observa- 
tions of Soviet ICBM tests should have been much 
more responsive to the ABM development. The 
sensors on the U. S. instrumentation ships 
should have been modified so they could collect 
the required data. Instead, large sums were 
used for devising questionable frequency and radar 
cross-section scaling relationships, for extensive 
target modeling, and for costly simulations which 
could not always be validated. 


Vast quantities of intelligence data were col- 
lected. There was enoughinformation on the 
composition and levels of Soviet forces, and an 
adequate design threat was eventually defined. 
However, data essential for design, develop- 
ment, and evaluation of critical target selection 
and tracking algorithms were lacking. They 
could have been obtained by observing Soviet op- 
erational tests. During the years 1967 through 
1972, data requirements were sent repeatedly to 
the Research, Development, Test and Evaluation 
(RDT&E) Directorate at Huntsville, who in turn, 
forwarded the requests to the intelligence agencies. 
With the advent of SENTINEL in 1967, UHFand S- 
band data were badly needed, yet UHF data re- 
mained scarce and incomplete andS-band data 
never were collected. 


Radar cross-section and trajectory data for 
supporting the ABM development were needed 
at the operating frequencies, polarizations, and 
resolutions of the MSR and PAR. There should 
have been a deliberate effort to collect data on 
RV wake characteristics at S-band, on tank 
breakup at S-band, and on all exoatmospheric 
objects at UHF. Wake and tank breakup data 
were collected later by the Meck MSR during 


live U.S. tests at Kwajalein, but these data were 
not totally adequate in the exoatmospheric regime. 


Hardware Design Implementation 
Hardware Design Release 


When SAFEGUARD was committed to Bee 
duction, schedules were quickly established for 
releasing hardware designs for manufacture. 
These designs were developed during the R&D 
program, and had to be baselined and trans- 
ferred from the R&D contract to the production 
contract. Hardware could then be manufactured 
' according to production schedules, but subse- 
quent design changes had to be made under for- 
mal production control procedures. These pro- 
cedures proved to be cumbersome in making 
changes to R&D designs not yet stabilized for 
the production system. It would have been 
more cost effective to start manufacture witha 
less rigid configuration control program, and 
to control changes under somewhat less strin- 
gent procedures. Then, after a transition 
period to allow for design stabilization, formal 
production procedures could have been intro- 
duced. 


A related problem was the Critical Design 
Review (CDR) required before the design could 
be released to production. The CDR was in- 
tended to allow the Army and Bell Laboratories 
to assure themselves that the design met system 
requirements and was ready for release to pro-~ 
duction. Since the equipment components had 
different lead times and were ready for release 
at different times, CDRs were held at various 
levels, on hardware ranging from magnetic 
apparatus to power supplies to power racks to 
subsystems. The CDRs should have been held 
only for large, very important items such as 
the MSR receiver, while lesser items should 
have been released prior to a CDR if required 
by production schedules. Army engineers 
could have been kept informed of the design 
status through the same type of regular contact 
with the design engineers that existed in the 
development program. 


Design Drawing Reviews 


In normal practice, the drafting organization 
and the responsible engineer spot-check com- 
pleted drawings for accuracy and then release 
them for manufacture. If errors are detected 
later, the responsible engineer is notified and a 
change order for the correction is prepared. 


In SAFEGUARD, an outside contractor re- 
viewed selected drawings submitted for release 
to manufacture. The intent was to avoid subse- 
quent change orders by assuring that the draw- 
ings contained no errors, In the early submit- 
tals, which came from reputable subcontractors, 
many errors were detected. More often than not, 
the errors did not affect manufacturing the pro- 
duct. Instead, they involved incomplete compli- a 


' ance with various drawing practices and format 


requirements. 


The result was numerous drawing rejections : 
and led to much effort being expended on addi- 
tional checks of drawings before they were 
released. Subcontractors completely checked 
all their drawings before releasing them. Then 
a prime contractor representative visited the 
subcontractor plant and conducted a further 
review. In spite of these repetitive reviews, 
drawing changes were still required when 
shops used the drawings for procurement and 
manufacture, because the shops found real 
problems associated with product manufacture. 
The expensive, time-consuming drawing 
quality audit, more often than not, uncovered 
only superficial problems. 


Drawing Designation 


An excessive number of drawings were gen- 
erated because of the initial requirement that 
each separate detail be on a different 11-million- 
numbered drawing. This volume of drawings is 
not required in commercial manufacture, nor is . 
it used in the Bell System. It is expensive both ES 


_ in dollars and in the time required for drafting. 


After much discussion of the cost of the pro- 
gram, the contracting agency gave some relief 
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by allowing contractors to include families of 
Similar parts on a single drawing, with designa- 
tion of a particular item by dash number. Cer- 
tain drawing details were also permitted on top 
assembly drawings only when the detail did not 
have a separate requirement, e.g., aS a spare 
part. If this relief had been extended to all 
subassemblies and been present from the start, 
considerable expense could have been avoided. 


Organization of Work 
System Responsibility 
Major responsibility for SAFEGUARD was 


concentrated in two Bell Laboratories groups — 
the SAFEGUARD System Design Department and 


_ the SAFEGUARD System Evaluation Department. 


The System Design Department generated re- 
quirements for the various subsystems based on 
the overall design. Implementing these require- 
ments was delegated to various project man- 
agers. The System Evaluation Department 
concentrated, at least initially, on evaluating the 
design, i.e., the requirements, rather than the 
implemented system. Neither of these depart- 


' ments was responsible for coordinating all im~ 


plementation activities and no other organization 
was assigned that responsibility. Therefore, 
coordination had to be supplied by the individual 
project managers and higher levels of manage- 
ment, 


The results were incompatibilities in system 
interfaces; significant delays in correcting sys- — 
tem requirements; delayed awareness of diffi- 
culties; and lack of adequate system evaluation 
of process design, program execution times, 
missile and radar interfaces, and the manual 
display and control subsystem. All of these 
areas were corrected by individual project man- 
agers but only at the cost of added effort and 
delay. 


In future projects, a system coordination 
group shoutd be established within the systems 
area. The group should be responsible for (1) 
coordinating all implementation, (2) interface 


compatibility, (3) coordinating system evalua- 
tion, (4) establishing and achieving project 
schedules, and (5) technical reviews of system 
design, system evaluation, implementation, and 
test programs. In addition, the group should 
ensure that lessons and techniques learned from 
the prototype system are transferred to the 
tactical design. The group should act as a 
technical contributor to all of the above ac- 
tivities, not just as an administrative staff 
responsible only for standards, schedules, 

and reporting. 


system Maintenance Organization 


Most SAFEGUARD designers recognized the 
importance of maintenance, and considerable 
effort was devoted to maintenance by individual 
design groups. However, there was no centra- 
lized guidance for maintenance design, and re- 
sults were less than optimum. For example, 
coordination of status reports and error respon- 
ses from the various subsystems could have 
been improved, and documentation and testing 
of the overall maintenance system could have 
been better. 


To get a better maintenance design, an organ- 
ization with the following charter should have 
been established: 

e Develop and maintain system and subsys- 


tem maintenance requirements consistent 
with the system availability goals 


e Guide selection of maintenance techniques 
and tradeoffs in each hardware and soft- 
ware design area 


e Guide development of operations and 
maintenance procedures and the associ- 
ated documentation 


e Develop plans for system-level tests of 
maintenance tools (especially for real- 
time operations). 


Like any other element in a system, main- 
tenance requirements will continually evolve as 
system design evolves. These requirements 
and their implementation are highly dependent on 
system Availability/Reliability goals (which 
themselves may change with time). Thus, it is 
important to keep system maintenance require- 
ments up to date and appropriate to the system's 
primary mission, | 


Unified Responsibility for Design and Development 


The success of a complex system, including 
its ability to operate on demand, depends on the 
availability and reliability of every essential 
element init. Thus, "system" cannot mean 
portions of an overall complex, partitioned to 
fall within predetermined administrative juris- 
dictions; no element is separable if its failure 
would prevent the system from fulfilling its 
operational task. In SAFEGUARD, the critical 
complex includes support facilities, such as 
power, environmental cooling, and communica- 
tions, as well as the technical equipment. A 
' diesel engine, a cooling system fan, or a data 
transmission circuit may be just as crucial to 
battle readiness as a data processor unit or a 
missile guidance set. 


An important reason for the success of the 
SAFEGUARD project was that responsibility 
for the design and development of essentially 
the entire system was concentrated in one 
organization. The only exceptions, the Tactical 
Support Equipment (TSE) and the communication 
system, caused significant problems. Also, 
there were difficulties because responsibility 
for training, supply, and maintenance was 
separated from the development responsibility. 


Because the communication system's techni- 
cal direction and funding were under the 
SAFEGUARD Communications Agency (SAFCA), 
not the SAFEGUARD System Command (SAFSCOM), 
issues which could not be settled mutually had to 
be resolved by the SAFEGUARD System Manager. 
As a result, some problems were ignored and 
others took more time and effort to resolve than 
necessary. There was also a tendency to insu- 
late communications suppliers from Bell Labora- 
tories designers, probably to minimize the num- 
ber of design changes. But this impeded liaison 
on the working level, leading to poor under- 
standing of how the communications interfaces 
worked and ultimately to interface problems. To 
try to coordinate communications system design, 
frequent meetings were held (typically monthly). 
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These meetings used up time and manpower and 
required attendance by many agencies and com- 
panies, but they were no substitute for direct 
working-level contact among designers. 


With the TSE, the prototype test site's out- 
standing success in meeting objectives would 
have been impossible without unified authority 
for operating and maintaining both technical and 
support facilities. Many problems surmounted 
only with difficulty (e.g., the problems with the 
Meck power plant) could have been lessened or 
eliminated if such authority had started earlier 
during design, procurement, and installation. 

At the tactical site, placing more of the admini- 
strative responsibility for TSE with the Weapon 
System contractor would have allowed more 
efficient operation with fewer problems and 
delays. During a demanding test and installation 
program, it is impossible to attain either ade- 
quate speed of response or 2 common view of 
priorities with divided authority. Hence, unified 
responsibility for the total system, including 
both technical and support facilities, is ex- 
tremely important. 


Management 
Subcontractor Arrangements 


The size of the development required that 
design work be assigned to subcontractors and 
responsibilities be shared with government 
agencies. The production phases required de- 
cisions as to which potential suppliers should 
receive contracts. The success of the project 
attests that almost all of the arrangements with 
subcontractors and agencies were quite success- 
ful; however, here too, there were also some 
difficulties. 


Arrangements involved a wide range of sub- 
contractors and items. For example, Martin 
Marietta for development and manufacture of 
the SPRINT missile, IBM for software develop- 
ment, a manufacturer other than the design 
agency for production of missile motor cases, 
the Atomic Energy Commission (AEC) and 
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its subcontractors for design and manufacture 
of the warheads. 


Subcontracting work successfully requires 
attention to things which are, for the most part, 
obvious. However, these matters are so vital 
that elaboration is in order. 


Consider first the design responsibility that 
the prime contractor gives to a subcontractor. 
Requirements must be consistent, controlled by 
the prime contractor, and understood well by 
each party. Note that requirements need not be 
perfect and that during the development they will 
change. 


Ways to evaluate progress should be care- 
fully determined before the subcontract goes 
into effect and then followed tenaciously so that 
real status is known by the technical project 


' management on each side of the interface. Be- 


cause requirements do change, progress will 
necessarily be assessed against a moving base. 


Contracts and other formal arrangements 
must not impede the flow of information or in- 
hibit changes. They should encourage progress, 


not stifle it. 


The method for subcontracting the software 
development was somewhat unique. Work was 
broken into a collection of well-defined relatively 
small tasks that could be handled between prime 
and subcontractor at a first-line supervisory 
level. Progress was rated monthly by the prime 
contractor's supervisors, and the rating deter- 
mined the award to the subcontractor on the 
cost-plus-award-fee contract. 


Extreme care should be taken when it seems 
desirable to award manufacturing contracts di- 
rectly from the government to other than the 
subcontractor who did the original design. If 
this is done with items that are intricate or use 
new technology, the expertise of the original 
designer is lost. Also, the expertise of the 
prime contractor in making system tradeoffs 
with respect to the item is lost. This kind of 
"breakout" should be used only where items are 
quite stable. 


When an item involves multiple government 
and contractor agencies, such as development 
of a missile warhead, it is obvious that the re- 
sponsibility and requirements assigned to each 
agency must be considered with care for all 
phases of the program. With warhead develop- 
ment, the designs and interfaces could have 
been simplified if complete responsibility, in- 
cluding the adaption kit, had been delegated to 
a Single agency. Again, communication, flexi- 
bility, and constant monitoring are required if 
development is to be effective. 


In summary, on a big project, assigning 
work to subcontractors is absolutely necessary, 
can be handled successfully, and requires 
constant management attention. 
Cost-Plus-Award-Fee Contracting for Software 
Development 

On the SAFEGUARD project, a large part of 
the software development had to be subcontract- 
ed. This posed a difficult problem, because 
the software had to be of high quality and be 
delivered on time, in spite of many requirement 
changes and complex interfaces. With all these 
constraints, development problems had to be 
sensed rapidly and accurately, and acted on 
promptly and effectively, or else development 
would get out of control. The key need was for 
close and continued attention to the subcontracts. 


To attract and hold the attention of subcon- 
tractors, the profit from the job was determined 
by job performance. In principle, the Cost-Plus- 
Incentive Fee (CPIF) contract does this by ad- 
ding a fee determined by applying 2 previously 
agreed-upon formula to objective measurements 
of cost and/or performance and schedule events, 
on completion of the work, Incentive fee con- 
tracts were used effectively on many parts of 
the SAFEGUARD development, However, CPIF 
contracts are difficult to use for developing 
complex software, because it is impracticable 
to set up predetermined performance goals to 
measure against the final product. 


The Cost~-Plus-Award Fee (CPAF) contract 
overcomes this handicap by providing that the 
fee be determined in real time and based on the 
subcontractor's performance as judged sub- 
jectively by the customer. The CPAF contract 
is a cost-reimbursable, level-of-effort arrange- 
ment in which the fee to be paid for each (pre- 
determined) period is based on the customer's 
unilateral judgment of the supplier's perfor- 
mance, measured against previously agreed- 
upon subjective performance criteria. The fee 
is not subject to change. This type of contract 
had been used to some limited extent by NASA 
and the Navy, but was not in general use. 


The SAFEGUARD CPAF subcontract for 
software development was in operation for 
more than four years, involved up to about 
800 people, and as of October 1974, covered 
over $130 million. It has been a good vehicle 
for dealing with a large, complex, dynamic 
problem, where the customer needs as gooda 
job as he can get, and ontime. This type of 
contract requires good faith between customer 
and supplier, and substantial monitoring and 
evaluation. The format encourages good cus- 
tomer-supplier communications and the manage- 
ment involvement that is necessary for success- 
ful performance. The improved visibility of 
problems makes it possible to solve them 
quickly. 


Incentive Contracts 


Traditionally, during large, long-term 
development projects, principal events are used 
to demonstrate the achievement of significant 
milestones. Most military contracts now in- 
clude incentives — contractually established 
monetary awards (positive or negative) —for 
achieving certain events. Many such incentive 
events were scheduled and successfully achieved 
during the SAFEGUARD development. The 
event system was valuable in demonstrating that 
performance requirements were being met and 
in encouraging project-wide planning. Further- 
more, the events were interim goals, each 
clearly supporting and contributing to ultimate 
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success, sothat morale was boosted by a sense 
of achievement when goals were reached. The 
monetary awards associated with incentives 
can provide these same important advantages. 
However, requiring the completion of certain 
events in the contract ensures that they are 
scheduled, while the monetary awards ensure 
proper management attention. | 


Incentive contracting has potential disadvan- 
tages as well as advantages. Based on the 
SAFEGUARD experience, the following cautions 
and recommendations are given. 


e Incentive contracting of performance, 
schedules, and.costs is beneficial only if 
firm performance requirements, schedules, 
and costs can be negotiated. The effec- 
tiveness of incentives in state-of-the-art 
development projects is severely limited, 
because milestones cannot be easily identi- 
fied and costs are difficult to negotiate. 
Incentive contracting loses many of its 
advantages if the contract has to be re- 
opened because of major changes in re- 
quirements and schedules. If this 
happens, a contractor fully aware of 
his problems will make every effort to 
"get well" as part of the change negotia- 
tions. 

e Incentive events must be significant, well- 
defined, and carefully chosen in-line 
milestones that must be reached to com- 
plete the project. They should be major 
performance or schedule achievements 
that require the success of many lesser 
milestones. Consequently, they should 
not be ends in themselves, and achieving 
them should not divert a substantial por- 
tion of the project's resources. Further- 
more, if conditions change and an incen- 
tive event no longer serves its original 
purpose, the military and contractor. 
should immediately negotiate to remove 
or modify it. To help avoid excessive 
diversion of resources, limit the time 
allowed for demonstrating an event. 


e Establish incentive monetary awards only 
for results that are useful to the project. 
For example, an award should not be 
given for delivering a unit before it can 
be used. Similarly, an award should not 
be established for greater power output | 
than other elements can handle or the 
system can use. Sound technical and 
business management skills are needed to 
work out the most challenging incentive 
events for a contractor and to assure 
overall success. 


@ Schedule incentive events infrequently to 
keep them significant and to allow prepara- 
tions to be integrated with normal work. 
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For projects several years long, schedule 
incentive events no more often than every 
three to six months for each major sub- 
system. Events should be scheduled 
approximately eight to twelve months ahead 
of their demonstration date. 


e Well in advance of performance incentive 
events, agree upon a reasonable time period 
or "window" for making the performance 
measurement. Failure to reach the event 
within that time peried should result ina 
maximum penalty. 


e Well in advance of a specific incentive 
performance, prepare a "catalog" to cover 
how the test is to be carried out, what 
instrumentation is required, what consti- 
tutes a success or failure, and how to 
score a failure from causes outside the 
contractor's responsibility. Murphy's 
Law states that if there is a chance for 
misunderstanding, it will occur. There- 
fore, careful attention should be given to 
preparing the catalog to make sure all 
factors are considered. 


Reporting Systems 


In large programs such as SAFEGUARD, 
reporting of technical performance, schedules, 
and costs is required for internal control and 
for informing the customer of program status. 
While these program aspects are interdependent, 
they are handled by separate reports and will be 


_ discussed separately. Each reporting system 


evolved over the years to meet changing program 
needs. 


The goal of the reporting system is to convey 
the right level of timely information while con- 
serving the effort required to prepare the report. 
Early in the program, technical progress reports 
were submitted every six months. They were 
difficult to prepare and, because of the interval 
covered, contained much outdated information 
which, while of historical interest, was not of 
immediate concern to program managers. Since 
then, and throughout most of the SAFEGUARD 
program, Bell Laboratories prepared monthly 
technical progress reports for the Army, sub- 
mitting them by the 20th of the following month. 


These reports had enough detail to present 
the current technical status for program man- 
agement and to serve as a permanent record of 
job progress. Additional details were trans- 
mitted less formally by individuals, at meetings, 
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and via memoranda. The monthly technical 
report also gave Bell Laboratories managers 
reasonably detailed information, not only 
about their own activities, but also about other 
phases of the program — a necessary part of 
the program coordination. 


Several weekly TWX reports were sent be- 
tween different project locations. These were 
uSeful because they were timely and concise. 


For reporting schedules, Bell Laboratories 
evolved the SAFEGUARD Program Schedule 
Charts, which showed program milestone events. 
They were submitted to the Army at monthly 
intervals and were also distributed within Bell 
Laboratories. These significant benchmarks, 
plotted against a calendar grid, supplied the 
information needed to assess program status 
and to coordinate various program activities 
with other Army units, such as the Corps of 
Engineers. 


These schedule charts tracked a limited 
number of easily recognizable milestones. Each 
represented substantial program effort and was 
essential to program completion. The charts 
were backed up by detailed schedules kept by 
each manager of a major subsystem. [If the 
dissemination of the schedule charts raised 
questions, the manager could supply the answers 
from his more detailed schedules. The schedule 
charts retained all milestone dates once they 
were established. Where changes were required 
and new targets agreed on, the old dates were 
retained so that the slippages were easily visible. 
The dates when events were accomplished, 
either early or late, were also shown. 


The level of detail in these SAFEGUARD 
Program Schedule Charts proved to be entirely 
adequate for the overall program management, 
They were much preferred to schedule systems 
that showed thousands of events, each involving 
a small amount of effort covering a short time 
interval. Had such a system been used, admin- 
istrative costs alone would have been excessive. 


Cost reporting used a system that summar- 
ized detailed cost information to facilitate 
management review and decisions. The report- 
ing structure showed costs for each subsystem 
with a further breakdown to some half dozen 
separate accounts. In each account, costs were 
shown in four categories: Engineering, Drafting, 
Subcontracts, and Other Direct Charges. At 
the beginning of each year, a spending plan was 
submitted with this same structure. The Finan- 
cial Management Report compared actual month- 
ly costs against the yearly plan, When neces- 
sary, the responsible technical manager sub- 
mitted a revised estimate of the costs for each 
remaining month of the year, with a concise ex- 
planation of the reasons for the revision. 


Although every effort was made to see that 
the Financial Management Report clearly set 
forth program status, an analysis of this re- 
port, together with any recommendations for 
redistributing funds, was furnished to the 
Army each quarter. The analysis and recom- 
mendations were essential to keeping top-level 
management informed. This reporting system 
and its level of detail were adequate for the 
successful financial control of the program. 


Managing the massive, highly complex soft- 
ware subsystem for SAFEGUARD presented new 
challenges, and new systems evolved to control 
it. In general, standardized reporting systems 
met internal project needs fairly successfully. 
They enforced a basic level of planning and — 
stimulated periodic surveillance of status. How- 
ever, different levels of management required 
different levels of detail, and their requirements 
often could not be satisfied by one report. The 
useful life of status information to managers who 
had to take direct action was much shorter than 
the shortest time it took to gather data for a 
project-wide report, prepare the report, and 
distribute it, Also, it was frequently difficult 
for managers to evaluate the accuracy of in- 
formation in written standard reports. Most 
managers relied on direct one-to-one discus- 
sions with their subordinates to obtain informa- 
tion that required immediate action. 
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One of the reporting systems employed was 
the computerized Management Reporting System 
(MRS). Its data base, which had some 3000 | 
items, contained scheduled start and completion 
dates of various Significant activities. Depen- 
dencies between the various items were indica- 
ted. Information was updated each month and 
published as a standard report. The time needed 
to gather, process, and publish information was 
approximately three weeks. 


In addition to the general positive and nega- 
tive attributes of these standard reports, two 
aspects of MRS stand out: 


1. Schedule problems between different 
project areas were highlighted. 


2. Special weekly reports could be hand- 
tailored for one person, In some cases, 
they were more useful than the standard 
monthly report. 


In software development, numerical meas~. 
ures of progress, such as the number of rou- 
tines coded and the number of trouble reports 
corrected, were usually found to be of limited 
value. The quantities counted or measured 
were not uniform with respect to the amount of 


‘work involved or its importance. Consequently, 


it was very difficult to interpret their signifi- 
cance. 


Principal Events Reports 


The most successful report prepared for 
agencies outside Bell Laboratories was the 
Principal Events Report used for software 
development and system integration. It identi- 
fied a number of important concrete milestones 
and defined them in detail. Many of these 
milestones marked the completion of tests on 
various functional capabilities. Reports on 
such events were normally made by TWX within 
48 hours after their scheduled completion. The 
TWXs gave the dates for completion of late 
items, while follow-up TWXs were then sent on 
the rescheduled dates. The Principal Events 
Report, which was issued quarterly, described 
each item, gave its status, and accounted for 
previously completed events. This system of 
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reporting was extremely valuable to higher 
management, with the TWX reports particularly 
useful. 


Miscellaneous 
Alternative Problem Solutions 


There are often many alternative approaches 
to solving management or technical problems. 
In many cases it was more effective to choose a 
good alternative and then carefully implement it 
rather than spend a lot of time trying to optimize 
the choice among alternatives. Often, the initial 
analysis of a problem leads to a complex solu- 
tion which may not really be essential to imple- 
ment, and further review may lead to solutions 
which are "elegant" in their simplicity. Project 
managers should repeatedly challenge their en- 
gineers to seek simple solutions to design prob- 
lems. 


In several instances, limited budgets led to 
modified requirements and less complex and 
costly designs. Constraints on dollar resources 
can establish hard priorities and stimulate sim- 
pler solutions. This was well expressed by 


- Robert Townsend:* 


"A tight budget brings out the best 
creative instincts in man. Give him 
unlimited funds and he won't come 
up with the best way to a result, 
Man is a complicating animal. He 
only simplifies under pressure. 

Put him under some financial 
pressure. He'll scream in anguish. 
Then he'll come up with a plan 

which, to his own private amazement, 
is not only less expensive, but also 
faster and better than his original 
proposal, which you sent back." 


independent Testing Operations 


Inevitably, the time allocated to hardware 
installation and testing in any big system will be 
compressed as much as possible, This obvi- 
ously makes it desirable to parallel operations. 
However, in complex systems the required test- 
ing is also_unusually complex. When the system 


*Up the Organization (New York, Knopf, 1970). 
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is controlled by a centralized data processor, 

as in SAFEGUARD, the tests must usually in- 
clude the data processor, so the amount of par- 
allel testing possible is limited. Using relative- 
ly inexpensive minicomputers for much of the 
early testing can alleviate this problem. The 
savings from reduced installation and test time 
may be much greater than the costs of using 
minicomputers. One example of this approach 
concerned the Remote Launch Equipment (RLE). 


A one-farm, two-missile prototype of the 
RLE had been tested at Kwajalein, and the 
tactical hardware had been given limited string 
testing in North Carolina. The first attempt 
to operate in a full four-farm configuration, over 
actual data links with a full complement of ac- 
tive launch cells, took place during installation 
and testing at Grand Forks, North Dakota. 
Therefore, rather complete tests were neces- 


sary. 


During planning, it was predicted that when 
unit tests of RLE were complete, the ensuing 
system tests would require full-shift operation 
on most working days for several months. 
Schedule conflicts precluded timely completion 
of RLE system testing under control of the 
SAFEGUARD data processor. So a minicom- 
puter, with a hardware interface unit and appro- 
priate software, was temporarily installed at 
the tactical site to simulate the data processor. 
A similar installation had been used success- 
fully at Meck Island during the prototype remote 
launch tests and at North Carolina locations for 
string tests. With the minicomputer, the tests 
were completed on schedule. The minicomputer 
also served as a diagnostic tool for difficult and 
complex design troubles, and surpassed the 
power of the Maintenance and Diagnostic (M&D) 
System and the built-in self-checks. 


Communications for System Tests 


Testing a system in a building as large and 
complex as the Missile Site Control Building 
(MSCB) requires a lot of internal communication, 
and running tests in several buildings at sites | 


many miles apart is even more difficult. Care- 
ful planning is required for the communication 
system used during this testing. Instead of 
planning Separate communication systems for 
testing and tactical operations, one Should use 
the tactical communication system for testing 
wherever possible. However, nontactical cir- 
cuits are required to support testing, particularly 
in the installation and early test phases. These 
circuits should be constructed around the basic 
framework of tactical communications. Such an 
approach allows an early evaluation of the tactical 
communication system, avoids the disruptions of 
switching from the testing to the tactical com- 
munications, and minimizes the cost of circuits 
required only for testing. This approach is best 
carried out by a single organization responsible 
for planning both the tactical communications 

and the testing communications. 


importance of Test Planning 


It is universally accepted that good test plan- 
ning is important to development of any system. 
For large real-time systems it may well be the 
most important part. Even when its importance 
is recognized, it is very difficult to make the 
best use of testing at each development stage. 
Although test planning in SAFEGUARD was prob- 
ably more extensive than in any other Single 
development, additional effort would have been 
worthwhile. 


Many aspects of system development are 
strongly affected by the way tests are planned. 
The following are some of the most important 
effects recognized during the SAFEGUARD 
development, 


e One of the first issues settled was the 
time allocated to system testing. Im- 
mediately after the deployment decision, 
the design, production, site construction, 
and installation had to be scheduled and 
coordinated. Inevitably, the planners 
wanted to achieve the earliest possible 
completion date. Because it had a direct 
impact on the completion date, one of the 
most important decisions was how much 
time to set aside for final system integra- 
tion and test, It was impossible to list 
the problems which would be faced, and so 
it was very difficult to specify the tests 
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and, therefore, a definite time interval. 
For a system as complex as SAFEGUARD, 
the test interval will be between one and 
two years. Hence, the question is an 
important one. 


With any system, the amount of testing 
(and therefore the testing time) is related 
directly, although in a very complicated 
fashion, to system quality. Reducing the 
amount of testing reduces quality, but 
increasing the amount of testing does not 
increaSe quality at a fixed rate. It is not 
at all unreasonable to limit the time and 
resources used for testing if it is recog- 
nized that doing so will have definite im- 
plications on quality. 


For a system like SAFEGUARD, a limited- 
quality product can be accepted at the out- 
set of system life. Quality can then be 
constantly improved as testing continues 

at site installations and at the prototype 
development site. This additional testing 
does not have the same incremental cost 
in resources as testing during the develop- 
ment cycle, because most of the resources 
must be available for routine operation. 
For a multisite deployment, the first 
installation need not have the ultimately 
desired quality, because tests can continue 
while other sites are installed. Conse- 
quently, the system test program can 
extend well beyond the first installation 
and perhaps even well beyond the final 
installation in a multisite deployment. The 
planners must understand the implications 
for system quality before the entire test 
plan is finished. 


Another issue which has a major impact 
on schedule is the number of software 
tests. For a complex system, it is always 
possible to conceive of an infinite number 
of tests, so test planners must first es- 
tablish a finite set. Next, project man- 
agers are rarely willing to allocate the 
time required for all the tests, so the 
problem is to select some subset of tests 
which is in some sense optimal. In 
SAFEGUARD, as a first step, the major 
functional capabilities to be used in im- 
plementing the system were listed. Then 
this set of possible tests was analyzed to 
see how it exercised these functional 
capabilities. A choice was then made as 
to whether each capability was to be ex- 
ercised at all, nominally, or stressed 

to some extent. Then, a set of tests 
which implemented these decisions was 
chosen, and software test planning was 
pointed at that specific set. This approach 
Seems to have produced an adequate testing 
program. 


Detailed test planning for system integra- 
tion, based on experience with the proto- 
type MSR, began in 1970, four years be- 
fore Equipment Readiness Date (ERD). 
Planning considered test facilities (system 
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exerciser, threat tape generation) and the 
scope of testing (threat scenarios, traffic 
levels, system configurations and operating 
modes, threat excursions, confidence 
levels). Planning went on continually from 
1970 until ERD, but even then, late deliver- 
ies of threat tapes caused problems in sys- 
tem test and integration at the TSCS. Al- 
though it is probably impossible to avoid 
this, careful monitoring is essential to 
minimize such problems because their im- 
pact on schedules can be major. In test 
planning, it is important to consider the 
reliability of test facilities. It is tempting 
to reduce the reliability requirements on 
test equipment not required for tactical 
operation (e.g., tape drives), but lower 
reliability will inevitably lengthen test time. 


As test planning proceeds, there will 
probably be important interactions with 
system design. The earlier these inter- 
actions are recognized, the lower their 
impact on cost, schedule, and system 
performance. Consequently, specifica- 
tion of system requirements should in- 
clude requirements on the subsystems 
needed for testing. 


To illustrate such interaction, when 


_ design of the system exerciser was well 


along, a serious short-time peak developed 
in the data processing requirements to 
test the system at the desired traffic level. 
Providing this capacity by increasing the 
data processing hardware in the system 
exerciser would have been costly. By 
modifying the tactical system design, the 
requirements on the system exerciser 
were reduced and the necessity for the 
additional hardware was thus avoided. 

This design modification did restrict ac- 
tual system performance, although only 
very slightly. If the situation had been 
recognized later, it could have been re-~ 
solved only by much greater financial 
costs, disrupting the schedule, or limi- 
ting performance. 


Another major question is how much testing 
to do at each installation of the system. 
For SAFEGUARD, it was decided that each 
installation would be tested as thoroughly 
in all respects, including traffic level, as 
the system was during development. 
Hence, each installation was given a com- 
plete system exerciser facility, even 
though this involved substantial cost. Such 
a decision hinges on issues specific to a 
particular system. In the case of 
SAFEGUARD, the major issues were: 


1. Because the system might never be 
battle-tested prior to an actual 
engagement, it would be very 
difficult to determine whether its 
capability was deteriorating unless 
regular exercises could stress it to 
nearly its full extent. 
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2. Because crews must be ready to 
respond in minutes, if not seconds, 
it was important to exercise their 
responses, 


3. It was important to thoroughly test 
each site, because each was assem- 
bled from tens of thousands of sub- 
assemblies, drawers, and racks; 
there were millions of interconnec- 
tions and contacts and countless 
critical signal and timing interrela- 
tions. Ifa system could not be 
tested, it could not be made to work, 
or be kept working. 


e Defining the requirements for data reduc- 
tion of test results is important but diffi- 
cult. In a system as complex as 
SAFEGUARD, every test produces a tre- 
mendous amount of data, For testing to 
proceed at a reasonable pace, the tester 
must be able to look at just the portion of 
the data he's interested in and to obtain it 
in the most easily understood way. How- 
ever, early in the design he is unable to 
Specify in detail what data he needs or 
how he wants to see it. Thus, itis diffi- 
cult to build a data reduction facility by the 
time it is needed. In SAFEGUARD, this 
problem was approached by developing a 
series of data reduction systems. While 
this may be the only solution, it did re~ 
quire a very substantial effort. By apply- 
ing more pressure earlier in the design 
cycle, designers would be forced to agree 
on a set of requirements to reduce this 
total effort. 


Site for Prototype Testing 


The TSCS, a partial prototype of the PAR 
and MSR, was built for developing software for 
the deployed system. Earlier in the program a 
prototype data processing system had been pro- 
vided for developing the R&D software. Soft- 
ware could be verified with the TSCS only to a 
certain limit; because actual sensor and missile 
ground/interface equipment was lacking, the 
software development had to be completed at 
the field site. These R&D software deficiencies — 
were remedied in the TSCS by including sensor 
parts that had direct or critical timing inter- 
faces with the data processing system and selected 
parts of the missile ground and communications 
subsystem. Simulating responses for the sen- 
sor and other equipment is not a complete sub- 
stitute for the equipment itself. Responses have 
to be limited, and the simulation can introduce 
artificial timing problems. It is difficult to 


anticipate and design a simulation that has all 
the non-nominal and unexpected responses 
which can have serious impact on software op- 
eration. Simulation can only model the hard- 
ware, and the actual hardware may not conform 
to design intent. 


The choice of equipment for the TSCS proved 
to be effective, and software developed and tes- 
ted at the TSCS was brought up rapidly at the 
installation site. The software used to install 
the hardware at the site and later to maintain 
it was complete and of good quality. Not only 
did the high quality of this software allow the 
site installation to proceed quite rapidly, it 
also established a stable base for the more 
orderly development of the applications soft- 
ware at the TSCS. 


Software designed to control hardware not 
included in the TSCS could only be partially 
debugged and tested at the TSCS; testing was 
completed at the tactical site. There was not a 
great deal of this software, but it was clearly 
not as well checked out when it arrived at the 
- site. 


As software was developed at the TSCS, many 
hardware design and manufacturing problems 
were uncovered. Many of the problems were 
corrected before hardware was installed at the 
site and, in almost all cases, solutions develop- 
ed at the TSCS were available for site installa- 
tion before the hardware was needed for the 
system test program, 


The TSCS made site installation very effi- 
cient, so that the major efforts could be directed 
to solving problems unique to the site. Besides 
the efficiency gained in installation and check- 
out, the essential completeness of the prototype 
system-level problems could be identified and 
resolved. These problems, which included op- 
erational situations, were found long before they 
would have-been detected at the tactical site. 
Hence, system integration progressed rapidly. 
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Dissemination of Information 


It was hard for people to keep up with overall 
progress and problems of a job as complex as 
SAFEGUARD. To acquaint people with basic 
decisions and problems was essential, but it had 
to be balanced with the cost in time. For higher 
management, the need was amply met by month- 
ly half-day sessions covering technical status 
and project-wide problems. Such meetings keep 
subsystem project heads technically sharp about 
problems in their areas of responsibility — 
particularly when the top managers have the 
technical acumen to probe sensitive areas. 
Clerical and support personnel were kept in- 
formed by quarterly filmed progress reports, 
and the inevitable time lag was acceptable for 
this audience. For lower management, there 
were occasional briefings on problems closely 
related to their work, typically on an organiza- 
tional basis. Little in the way of information 
dissemination was done for nonsupervisory tech- 
nical people. 


More time should have been devoted to 
communicating with lower management and 
technical nonsupervisory personnel. The re- 
ports described earlier were good, but they 
were not always very current, could not be 
questioned, and frequently went unread. A 
better system might have been a half-day of 
several individual talks every month or six 
weeks, Information such as the following might 
have been communicated more effectively: 


e Problems already solved and reflected in 
decisions. This area is important because 
the decisions can affect people's work 
directly, and the problem-solving tech- 
niques may be applicable elsewhere. 


e Selected tough problems not yet solved. 
The payoff here is to encourage ideas 
from people who may be able to supply 
them. Also, a tough problem can some- 
times be alleviated by energetic action in 
another area. A difficulty is that groups 
are reluctant to discuss a problem not yet 
completely solved, lest it suggest in- 
competence, Management should en- 
courage the airing of such problems when 
necessary. 
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e Problems which have been formulated, 
but where solutions are not at all apparent, 
or are likely to be put off by the pressure 
of events. By raising these problems, 
they can be considered explicitly where 
they would otherwise have to be bypassed 
because of time pressure if left to upper 
management. 


Transfer of Personnel 


On a large development project, information 
must be exchanged between groups working on 
prototype systems and those working on produc- 
tion systems, and between groups preparing 
system requirements and those implementing 


them. The only feasible way to transmit most 


of this information, of course, is in writing or 
verbally. Some of this information exchange 
Should occur via transfers of people between 
organizations so that ideas and points of view 
can be incorporated into the work as fully as 
practicable. 


In the short term, transfers reduce the 
ability of the organization losing the personnel. 
However, this short-term cost is more than 
made up for by the long-term benefits of under- 
standing gained by the receiving organization. 


‘On SAFEGUARD, substantial numbers of people 


were transferred from prototype to production 
development organizations and between other 
groups during production development. Even 
more transfer of personnel would probably have 
been desirable, 


Timeliness of Problem Recognition 


It is important that problems be recognized 
quickly, that solutions be carefully planned, and 
that proper resources be applied. Some prob- 
lems which should be addressed early simply do 
not get timely attention. Others are addressed 
too early and inappropriately "solved," only to 
reappear later. 


For example, initial attempts to design 
SAFEGUARD data reduction packages foundered 
because of inadequate attention. They were 
aimed at testing and integration activities then 
several years away, while operating system 
design, language processors, algorithm 
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development, and guidance simulation were com- 
manding prime attention. This tendency to dis- 
count* problems perceived as remote inspace or 
time also hindered communication subsystem 
design. 


The impact of discounting should be minimized; 
it probably cannot be eliminated. Management 
should guard against it by periodic reviews of 
plans, forecasts, and current design activities. 

A comprehensive development plan is a decided 
help. Management should also restructure 
groups or departments as necessary to start 
early attacks on late-maturing design problems. 
Both of these schemes were tried and found 
useful. 


Staff Assistants for Managers 


Staff assistants, called Management Report 
Analysts (MRAs), were assigned to many 
second-level managers with project manage- 
ment responsibilities. The MRASs were, in 
general, college graduates with backgrounds in 
planning, scheduling, and budgeting. They were 
trained in the objectives, procedures, formats, 
and use of the SAFEGUARD Managing Reporting 
System (MRS) and could input up-to-date infor - 
mation into the MRS through their close associ- 
ation with their project manager. 


They also assisted the project managers by 
preparing and controlling budgets, administer- 
ing contracts, planning, scheduling, allocating 
space, arranging personnel moves, recording 
and following up on action items generated at 
project meetings, and performing numerous 
other administrative details. 


Almost all of the managers who were 
assigned MRAs were enthusiastic about their 
usefulness. By relieving Managers of many 
purely administrative functions, the MRAs made 
it possible for managers to concentrate on the 
more critical technical aspects of the job. 


*See H. A. Linstone, IEEE Spectrum, April 1974, 
for a discussion of discounting and forecasting. 


